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ABSTRACT 


We  consider  methods  for  the  approximate  solution  of  a  Tredholm  integral 
equation  of  the  first  kind,  namely 

u(t)  =  I  K(t,s)  z(s)ds,  t  «T 

's' 

where  K(t,  s )  is  a  given  kernel,  u(t)  is  given  foi  t  -  t  ,  t  ,  .  ,  .  ,  t  ,  and  z  , 

j.  «  n 

to  be  approximated,  is  an  element  of  a  reproducing  kernel  Hilbert  space  . 
We  assume  that  the  linear  functionals  A  ,  t  t  T  defined  by 

A  z  =  J  K(t,  s)  z(s)ds 
1  S 

are  all  continuous  on  W  _  and  we  treat  the  problem  of  approximating  N  z  =  z(s) 

R  s 

for  s  c  S  as  a  problem  in  approximating  the  continuous  linear  functional  N 

s 

by  the  continuous  linear  functionals 

(At,  t  =  •  •  •  » tn  1  • 

With  this  point  of  view  we  do  the  following: 

i)  give  formulae  for  approximate  solutions,  under  various  conditions 

ii)  give  pointwise  error  bounds  for  these  solutions 

iii)  give  several  examples  of  practical  reproducing  kernel  Hilbert  spaces 

iv)  show  that  the  method  of  regularization  of  Tihonov  and  Glasko  and  the 
method  of  statistical  estimation  of  Westwater  and  Strand  are  (equivalent) 
special  cases  of  the  theory  presented  here 

v)  show  how  to  choose  the  regularization  parameter  when  errors  due 
either  to  quadratures  or  experimental  measurement  is  non-negligible. 


ON  THE  APPROXIMATE  SOLUTION  OF  FREDHOLM  INTEGRAL  EQUATIONS  OF 

THE  FIRST  KIND 
by 

Grace  Wahba 


1.  Introduction 

1.1  Statement  and  Some  History  of  the  Problem 

We  are  interested  in  the  numerical  solution  of  a  Fred¬ 
holm  integral  equation  of  the  first  kind,  namely 

u(t)  =  /  K(t,s)  z(s)ds  teT  (1.1) 

S 

where  S,  T  are  intervals,  K(t,s)  is  a  given  kernel  on 
T  *  S  with  appropriate  properties  and  u(t)  is  given  for 
t  —  » ^2 ' ’ * " ' ^ r  t^GT • 

It  has  been  noted  by  a  number  of  authors  (see  for  ex¬ 
ample  17],  [11],  [14])  that  replacing  the  integral  in  (1.1) 
by  a  quadrature  approximation  and  inverting  the  resulting 
matrix  does  not  always  give  satisfactory  results.  No  matter  how 
large  n  is,  there  are  many  functions  z,  including  highly 
oscillatory  ones,  such  that 

u(t.)  =  /  K (t.  ,s) z (s)ds  ,  i  =  1,2,... ,n  (1.2). 

1  s  1 

A  discrete  approximation  of  one  of  these  functions  is 
obtained  by  the  matrix  inversion 
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technique.  Which  one  is  actually  obtained,  apparently  de¬ 
pends  more  on  the  quadrature  procedure  than  anything  else. 

Two  methods  in  the  literature  seem  to  have  resulted  in  satis¬ 
factory  numerical  examples.  The  first  method,  called  the 
method  of  regularization  of  Tihonov  112)  113]  and  studied 
experimentally  by  Tihonov  and  Glasko  [14],  goes  as  follows: 
Let 

(z,u)  -  /  (u(t)-u(t))2dtfa  /  (L  z(s))2ds  (1.3a) 

T  S  m 

A 

where  u  is  a  given  function  in  L2(T),  u,  depending  on  z,  is 
given  by 


u(t)  “  f  K (t ,s) z (s)ds ,  (1.3b) 

S 

a  >  0,  and  Lm  is  an  mt*1  order  linear  differential  operator 
with  continuous  positive  coefficients.  Assume  K(t,s)  is 
continuous,  and  that 

0  *  /  K (t ,s) z (s) ds ,  teT  (1.4) 

S 

implies  that  z(s)  =  0,  seS. 

Then,  it  is  shown  in  [13] ,  that,  for  every  fixed 
ueL2[T],  and  every  a  >  0,  there  exists  a  unique  2m  times 

A  A 

differentiable  function  z  *  zQ  which  minimizes  M“(z,u). 
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Tihonov  and  Glasko  114]  provide  an  argument  that  I,  defined 
by 


L  z  =  z ' 
m 


(1.5) 


fits  into  the  general  theory  if  K  >  0  on  T  x  S.  They  then 

a 

experiment  with  a  numerical  algorithm  based  on  finding  z  to 

minimize  m“(z,u)  as  follows.  Quadrature  points  {s.}k  , 

n  i=1 

s.eS,  and  {t.}n  ,  t.eT  are  chosen.  Let  t  be  the  left 

1  1  j=l  3  ° 

boundary  of  T,  define 


2  e  (Z  Z2  r  *  *  *  fZ,) 

U=  (uttj) ,u(t2) , . . . ,u(tn) ) 


and  zQ  =  0.  u  is  a  given  vector.  Define  m“<z,u)  as 


M?(z,u)  =  l  d . (  [  K  (t • ,s. )o. z.-u(t . 
1  j=l  3  i=1  Dill  3 


)  ) 


,  V  (ii+r5i’2 
+  a  I  - y 

i=°  (si+1-s,)2 


(1.6) 


where  {d. }n  ,  d .  >  0 ,  and  {j.}k  are  appropriately  chosen 
3  j=l  3  1  i=l 

quadrature  coefficients.  M°(z,u)  is  to  be  thought  of  as  a 
discretized  version  of  M^(zfu)  of  (i.  3a)  with  given  by 
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r 

i 


J1.5) .  Let  the  n  *  k  matrix  K,  the  n  *  n  (diagonal)  matrix 
D  and  the  k  *  k  symmetric  matrix  Q  be  defined  by  setting 

M®(z,u)  ■  (Kz-u) ’ D (Kz-u)  +  az'Qz  (1.7) 

and  matching  the  coefficients  of  z ,  u  and  ai  in  (1.6)  and 

A 

(1.7).  For  a  given  u,  the  vector  z  which  minimizes 
m“(z,u)  is  well  known  to  be  given  by 

z  *  (K*  DK+aQ) ~^K 1  Du  (1.8) 

It  can  be  verified  by  direct  manipulations  upon  the  matrices 
involved  that  another  formula  for  z  is 


Tihonov  and  Glasko  compute  z  from  (1.8)  as  a  discrete  approxi- 

A 

mation  to  the  function  za(s)  which  minimizes  m“(z,u).  They 
do  this  experimentally,  for  several  va’ ies  of  o,  by  beginning 
with  a  u  obtained  from  a  known  function  u(t)  satisfying 

u(t)  =  /  K(t,s)  z*(s)ds 
S 

* 

where  z  is  a  given  (smooth)  experimental  function.  For 
certain  values  of  a,  the  results  are  "good",  i.e.  the  com- 

A  A 

ponents  z^  of  z  satisfy 
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More  recently,  Ribiere  [8]  has  also  studied  the  method 
of  regularization.  In  [7],  (8]  and  (14]  the  experimental 
results  suggest  that  there  is  an  optimum  choice  of  a.  How¬ 
ever,  a  theory  making  precise  the  optimal  choice  is  apparent¬ 
ly  not  available. 

The  second  method,  discussed  by  Strand  and  Westwater 
[11]  and  called  "statistical  estimation"  of  the  solution, 
is  as  follows:  Let  the  n  x  k  matrix  K  be  defined  as  before. 
Le  t  Z  =  ,  ^2  ,  •  <  •  ,  and  c  =  be 

normally  distributed  zero  mean  random  vectors  with  {prior) 
covariance 


EZZ '  *  Q 


Ec(' 


fcZc  ’  =  0 


Let  U  =  , u2 , . - . , U  )  be  a  normally  distributed  random 

vector  defined  by 


U  =  KZ  +  c 


(1.10) 


It  is  well  known  that 


(1.11) 


_  -1-  -—1-  -1  — 

E(Z|U-u)  -  Q  K' (KQ  ■V+aD  )  u 

Strand  and  Westwater  replace  the  equation  (1.1)  with 
the  model  (l.io),  where  the  unknown  vector  z  is  replaced  by 
the  random  vector  Z  with  prior  distribution  N(0,Q~^),  and 
t  is  to  be  thought  of  as  a  "noise"  vector  independent  of  z 
and  having  the  distribution  N(0,aD~^).  The  estimate  for  the 
"random"  vector  Z  given  the  "data"  u  is  taken  as  the  right 
hand  side  of  (1.9).  Hence  the  method  of  regularization  and 
that  of  statistical  estimation  are  in  practice  the  same  for 
appropriate  choice  of  D  and  Q.  Strand  and  Westwater  per¬ 
formed  numerical  experiments  (again  beginning  with  a  smooth 
* 

known  solution  z  )  analogous  to  those  of  Tihonov  and  Glasko, 
and  again  obtained  "good"  results. 

1.  2  Purpose  of  This  NotJ  and  Basic  Assumptions 

Both  of  these  methods  can  be  embedded  in  the  general 
theory  of  the  approximation  of  continuous  linear  functionals 
in  a  reproducing  kernel  Hilbert  space.  The  overall  purpose 
of  this  note  is  to  demonstrate  this  statement  in  some  con¬ 
siderable  practical  detail.  As  a  byproduct  of  this  theory 
we  obtain  a  rationale  for  choosing  a,  to  minimize  a  certain 
error  bound.  We  also  obtain  pointwise  error  bounds  on  the 
solution  involving  a  Hilbert  space  norm  of  the  (unknown) 
solution  z.  These  results  provide  a  criteria  for  the  selection 
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of  a  good  Hilbert  space  in  which  to  operate.  Several 
practical  °‘>.amples  are  given. 

We  assume  the  function  z  to  be  an  element  of  a  repro¬ 
ducing  kernel  Hilbert  space  M  of  real-valued  functions  de¬ 
fined  on  S.  This  means  that  the  linear  functionals  N  , 

5 

seS,  defined  by 

Nz=z(s)  ,  zett  (1. 12) 

s 

are  continuous  for  every  seS.  This  is  just  what  is  needed  to 
establish  point  wise  error  bounds.  In  this  case,  for  VseS 
there  exists  an  element  <5  eS  for  which 

5 

<6  ,z>  =  z(s)  ,  Vze  m  .  (1.13) 

s 

If  we  define  the  kernel  R(s,s')  on  S  *  S  as 

<6s,6s,>  =  R<s's' )  (1.14) 

then  R(s,  s')  is  positive  definite.  Let  R  be  that  function 

so 

defined  on  S  whose  value  at  s  is  given  by 

R  (s)  =  R(s_,s)  (1.15) 

so  ° 

Ry  the  Moore-Aronsza jn  Theorem  (see  [2]),  to  every  positive 
definite  kernel  R(s,s')  defined  on  S  *  S  there  corresponds 
a  unigue  Hilbert  space  N  =  **  with  the  following  properties: 

K 
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1)  Rff  e  *  VsoeS 

2)  <z,R  >  *  z(s  )  #  zc  to  ,  s_eS 

s  _  o  o 

o 

Hence  6g  of  (1.13)  is  given  by 

6g  -  Rs  (1.16) 

and 

<R  .R  .>  -  R(s ,s ' )  (1.17) 

S  a 

Equation  (1.17)  is  the  source  of  the  terminology  "reproducing 

kernel".  The  elements  (R  ,seS>  clearly  span  u  . 

s 

We  make  sufficient  assumptions  on  K  tc  ensure  that  the 
linear  functionals  defined  on  »  R  by 

A  z  *  /  K(t,s)z(s)ds  ,  ze  W  (1.18) 

t  s  R 

are  continuous  for  every  fixed  tcT.  The  problem  of  estimat¬ 
ing  z(s)  for  a  particular  seS  given  that  ze  and  the  in¬ 
formation  of  (1.1)  ,  namely ,  A.  z  *=  u(t, )  ,  i  *  1,2, ... ,n  may 

ti  l 

then  be  viewed  as  that  of  approximating  the  continuous  linear 

functional  Ng  defined  by  (1.12)  by  the  continuous  linear 

functionals  {A.  }n  defined  by  (1.18). 
ci  i=l 
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To  discuss  the  problem  practically,  we  must  know  when 

defined  by  (1.18)  is  continuous,  and  how  to  find  its  representer 

nt-  The  answer  is  given  by  Theorem  (1.1)  and  its  corollary 

which  will  conclude  Section  1. 

Once  the  representers  (n,.  )n  are  known,  there  are 

ci  i«=l 

several  points  of  view  that  one  may  adopt  when  approximating 

one  continuous  linear  functional  by  several  others.  Perhaps 

the  simplest  is  as  follows.  Let  ii  be  the  subspace  spanned 

by  (n  }n  ,  and  P  the  projection  operator  onto  this  sub- 
ti  i=l  n 

space.  Suppose  is  n  dimensional.  Approximate  Ra  by 

P  R  .  Then  if 
n  s 

<nt.'z>  =  u<ti^  '  1  =  l»2,...n 


we  have 


<pnRg,Z>  =  <W>  =  (Pn2)  (3) 


=  (n*.  <s),n  (s) 

Zl  Z2 


u(t 


1>\ 


\  *  j 

U(V ' 


P  z  is,  of  course,  that  element  in  u  of  minimum  norm  which 
n 

satisfies  (1.2).  One  then  has  the  point  wise  error  bound 
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|z(s)-(Pnz) (8) I  -  I <2"pn*'Rs>!  "  I <z”Pn2'Rs“PnRs>l 


<  VPnRsll  ^  ||*||  IIVPnRs  II  (1.19a) 

where  llRs-pnRall  can  be  computed  for  any  fixed  s.  Hence  in 
solving  any  particular  practical  problem,  one  should  expect 
good  results  if  one  can  choose  a  reproducing  kernel  Hilbert 
space  in  which  the  norm  of  the  unknown  function  z  is  small. 

We  actually  generalize  this  point  of  view  in  two  ways.  First, 
if  ^  is  a  Hilbert  subspace  of  it  of  codimension  m  <  n,  then 
it  is  possible  to  approximate  RB  by  {n*.  }n  in  such  a  way 

8  H  i-1 

that  Rs  and  its  approximant  differ  by  an  element  in  In 

this  case,  it  is  shown  (in  Lemma  2.5)  that  the  right  hand 
side  of  (1.19a)  may  be  replaced  by 

llpi2llllpi(VVII  (1<19b) 

where  is  the  projection  operator  onto  and  Rg  is  the 

approximating  element.  Here  we  will  be  able  to  calculate 

I  IP1  (Rs”Rs)  I  I  '  and  one  ctiooses  611  w  and  an  *i  such  that  it 
is  believed  that  the  norm  in  A  of  P^z  is  small.  This  is  the 
geometry  of  the  approximation  of  continuous  linear  functionals 
used  by  Golomb  and  Weinberger  (4).  For  other  somewhat  re¬ 
lated  work  on  the  approximation  of  one  continuous  linear 
functional  by  several  others  see  [3]  and  [9] . 


-10- 


#990 


Futher  generalizing,  we  actually  consider  in  some  detail 

the  cases  where  u(t^)  are  known  only  up  to  some  error 

(experimental  error)  or  the  values  tk  (s)  of  the  representers 

ci 

n.  are  known  only  imperfectly  (quadrature  error) .  In  this 
*i 

case  the  error  bounds  analogous  to  (1. 19a)  involve  the  magnitude 
of  the  (experimental  or  quadrature)  error.  We  show  how  an 
approximate  minimization  of  these  error  bounds  leads  in  a 
natural  way  to  a  family  of  algorithms,  which  contains  the 
method (s)  of  regularization  and  statistical  estimation. 

Section  2  provides  statements  of  all  the  Hilbert  space 
lemmas  that  are  used  in  the  sequel.  They  are  generally  well  known. 
Since  they  may  be  proved  by  elementary  methods,  no  proofs  are 
provided.  Section  3  provides  examples  of  some  reproducing 
kernel  Hilbert  spaces.  Concrete  examples  are  fairly  hard  to 
come  by.  Section  4  presents  the  main  theorems  concerning 
approximations  to  the  solution  z  and  their  properties.  For 
concreteness  sake,  but  without  loss  of  generality,  the 
theorems  are  stated  with  respect  to  a  particular  reproducing 
kernel  Hilbert  space  which  is  related  to  the  method  of  re¬ 
gularization  as  discussed  in  (12],  [13],  (14].  Section  5 
discusses  tha  introduction  of  quadrature  error  and  appropriate 
ways  of  dealing  with  it,  and  a  rationale  for  choosing  a  is 
given.  In  Section  6,  the  precise  relationship  between 


statistical  estimation,  and  the  approximation  of  continuous 
linear  functionals  is  described,  and  it  is  shown  how  in 
general  one  obtains  the  sane  numerical  result  from  the  two 
approaches . 

We  conclude  this  Section  with  Theorem  1.1  and  its 
corollary. 

1.  3  A  Preliminary  Theorem 

Theorem  1.1.  Let  A  be  a  continuous  linear  functional 
on  #  R,  and  let  ipe  AR  be  defined  by 

Ai  “  <ip,z>  ,  z e  J*R 

Then  i|»(s),  seS,  is  given  by  the  formula: 

iMs)  -  AR  (1.20) 

S 

Conversely,  let  A  be  a  linear  functional  defined  on  i*R  and 
suppose  that  the  function  tp  defined  in  s  by 

ip(s)  -  ARa  (1.21) 

satisfies  (pe  aR.  Then  A  is  continuous. 

Proof.  Since  (p(s)e  (Ms)  »  <'I»,R_>  ■  AR_.  Conversely, 

■  Jt\  s  s 

if  <pe  then  the  continuous  linear  functional  A  defined  by 

Is 
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Az  =  <ty,z> 


(1.22) 


coincides  with  A  on  the  span  of  {R _,seS),  since  Xr_  ■  ^(s). 

5  S 

But  the  span  of  {R.seS}  is  &D. 

S  K 

Corollary.  Let  K  be  such  that 

/  /  K(t,s)R(s,s,)K(t,s' ) dsds '  *  e2(t)  (1.23) 

s  s 

is  well  defined  and  finite  as  a  Riemann  integral  for  each 
fixed  teT.  Then  the  linear  functional  A^.  defined  by 

A. z  =  /  K(t,u)z(u)du  ,  ze  (1.24) 

t  s  R 

is  continuous  on  #R  and 

Afcz  =  <nt,z>  ,  teT  ,  ze  *R 

where  nt  e  ^R  is  defined  by 

tk(s)  =  A.  R  =  /  K(t,u)R  (u)du  ,  (1.25) 

r  c  s  s  s 

and  | | A t | |2  =  e2(t) . 

Proof.  The  hypotheses  on  K  imply  that  there  exists 
(for  each  fixed  t) ,  a  triangular  array 
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811 

812'822 


slk,a2k' * ' ‘ ,8kk 


such  that  the  Riemann  sums 


k-1  £-1 


ih  K(t,9ik>  R,Sik'Sjl,Ktt'ajJlJ  *si+l,k“sik*  (aj+l,£-sj^, 


converge  to  02(t),  as  k,  £  •*  <»,  max|s^+^  k~s^k|  But  then, 

using  (1.17),  it  follows  that  the  sequence  n^k^ ,  k  «  1,2,..., 
defined  by 


ntk>  "  K(t,sik}  (si+l,k*sik)Rsik  k  "  1'2'*** 

is  a  Cauchy  sequence  in  WR,  ||n^kM|2  ■*“  92(t),  and  n£k* 
converges  pointwise  (in  s)  to  nt  given  by  (1.25),  which  is 
therefore  in  ftR. 


2.  Hilbert  Space  Lemmas 

In  the  lemmas  of  this  section  we  have  the  following: 
m  <  n,  ft  is  a  Hilbert  space. 


ft  .  ft  ffl  ft, 

0  1 
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where  ilQ  is  an  m-dimensional  subspace,  “  W  Q  and  PQ 
and  P^  are  the  projection  operators  onto  and  re¬ 
spectively.  are'  ^  each  lemma,  n  elements  in  W 
satisfying  the  following  two  conditions 


(i) 

{P0ni}i=l 

span 

(ii) 

<Vi>i-l 

are  linearly  independent. 

Let 

P1ni  *  5^  ,  i  =  1,2, ... ,n 


let  ^ ) v»»l  he  a  ifixed^  orthonormal  basis  for  and  let 

X  and  l  be  the  m  *  n  and  n  x  n  matrices  with  entries  de¬ 
fined  by 


fx] ui  =  <«J>lJ.ni>  ,  y  =  1,2. ..m 
ell ij  =  <Ci,Cj>  i»  i  ■  z-  •  -n 


(2.1) 


Conditions  (i)  and  (ii)  guarantee  that  X  and  ][  are  of  full 
rank.  We  let  <p ,  £  and  n  be  vectors  of  m,  n  and  n  elements 
of  a  respectively,  given  by 


<t>  “  ($1  *  $2  '  *  *  * 

£  “  (C1,52»* • 
n  =  (hj ,n2,. • -nn) 


1 


i 
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and  u  be  a  given  vector  of  real  numbers, 

G  -  (uru2,...un). 

Lemma  2.1.  The  solution  to  the  problem:  Find  ze  4 
to  minimize  <P^z,P^z>  subject  to  <z,n^>  ■  u^,  i  -  1,2, ...n, 

A 

is  unique,  and  is  given  by  z, 

2  ■  <xX’lx'  .  (2.2) 

Lemma  2.2.  Let  6Qe  »  be  given.  The  solution  to  the 
problem:  Find  ye  a  of  the  form 

y  -  Jl  <=i\  <2-3> 

where  c  »  (c^,c2 , . . .cni  is  a  vector  of  real  numbers,  to 
minimize 

! !pi(50_y) ! !2  (2*4) 


subject  to 


MVV^M2  »  0  <2.5) 

A 

is  unique  and  is  given  by  6C, 
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(2.6) 


<5q  -  $  (x^'1x'  )~1xl  (xj"1x,)"1xj"1)n‘ 


where  4>  and  X  are  the  vectors  of  real  numbers  (depending 
on  6q) ,  given  by 

4*  =  ( <^0 '  ^*1^  '  ^o '  <^2>  '  *  ’  *  ^o r  ^m^ ) 

(2.7) 

X  =  (<50*5i>'<‘50*^2>  '  ■  *  *  '  •  •  <6o'^n>^  ' 

* 

Let  the  bounded  linear  operators  A  and  A  be  defined 
by  (2.2)  and  (2.6)  respectively  as 


Az  =  z 


if  ui  =  i  *  l,2,...n,ze  # 


* 

A  6 


o 


6  e  # 
o 


* 

Inspection  of  (2.2)  and  (2.6)  reveals  that  A  and  A  are  each 
idempotent,  and  are  adjoint  to  each  other,  that  is: 

Lemma  2 . 3  If  z  is  of  the  form  of  the  right  hand  side 

A  ' 

of  (2.2)  for  some  u,  then  z  =  z. 

Lemma  2 . 4 


<z,6  >  =  <z,6  >  =  <z,6  >,  z,6  e*  (2.8) 

o  o  o  o 
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Lemma  2 . 5 


/\  a 


<*-z,«c>*  -  <z-z,6o-50>z  <  ||P1(z-z)||  ^(fi^ 


<  |  |Pl=:!  |2|  |P1<<50-«0)  f  I2  (2.9) 


Furthermore,  a  calculation  gives 
Lemma  2 . 6 


ilVVVIl 


2 


<P , 6  ,P. 6  > 
1  o  1  o 


+  (i-cr1x,j  (xjrvr^-ir1*')  * 

(2.10) 

where  and  %  are  given  by  (2.7). 

Lemma  2.7  Let  V  ®  {v^}  be  a  non-negative  definite 
r.  *  n  matrix  with  ijth  entry  of  v”1  given  by  v^.  Then  the 
solution  to  the  problem:  Find  ze £  to  minimize 


M(z) 


i,  j=l 


(<z»ni>-ui)v13 (<z,0j>-u^)  +  <PIz,P1z> 


(2.11) 


is  unique,  and  is  given  by  z, 


<f>  (XS-1X‘ 


)-1XS_1u' 


+  c  (s-1-s"1x' (XS' 


)“lXS~1)u' 
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I 


where 


S  *=  l  +  V. 


(2.13) 


Lemma  2 . 8  Let  be  n  elements  in  ii  satisfying 
<ei'ej>  =  vi-<'  and  let  V4  be  9*ven*  T^e  solution  to  the 
problems  Find  ye  w  of  the  form 


n 


y  “  I  d-i  (ni+e, ) 
i=l  1  1  1 


(2.14) 


where  d  ■  (d^,d2 , . . .d^)  is  a  vector  of  real  numbers  to  be 
found,  to  minimize 


I \&Q  ~  l  di^±\ I2  +  II  l  dieil I2 

°  i=l  1  1  i=l  1  1 


subject  to 


(2.15) 


n 


V{o  -  h  Vi*  I  I  -  0 


(2.16) 


is  unique,  and  is  given  by  6q, 


5q  =  $  (XS_1X’ )“1XS-1(n+e) '  +  l  (S_1-S“1S'  (XS-iX' )  1XS~1)  (n+e) 

(2.17) 


where  <J>,  £,  depending  on  6q  are  given  by  (2.7),  S  is  given 
by  (2.13),  and  the  vector  of  functions  (n+e)  is  given  by 
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i 


ri+e  -=  (n1+E1>n2+E2' ‘  ‘ ‘nn+en) 


(2.18) 


Lemma 


2.9  Let  z  satisfy  <z,n^+e^>  **  u^,  i  *  1,2,. ..n. 


then  if  z  is  given  by  (2.12)  and  5Q  by  (2.17)  ,  then 


<z  ,6  >  =  <z,6  > 
o  o 


(2.19) 


Lemma  2.10  For  d  *  (dL ,&2 , . . . dn)  any  vector  of  real 
numbers,  and  z  and  z  as  in  Lemma  2.9,  6Q  as  in  (2.17) 

<z-z,60>2  =  <z^0-^0>2  <  2 1 1  !  z  |  |2(|  ISQ-J^jhil  !2+l  I  iI1diEi 

(2.20) 


-)  ] 


If  eie  *  j_,  i  =  1 , 2 , . . . n ,  then  6 Q-60  e  and 

<z-z,6q>2  <  2  [  |  jPj^l  |2(|  I  60'’iIidiT1il  |2+l  li|1dieil  I2)1 


(2.21) 


3 .  Examples  of  Reproducing  Kernel  Hilbert_ Spaces 
3.1  Generalities 

Let  S,  f2  each  be  the  real  line  or  a  closed  subset, 
let  G(s,u)  be  defined  on  S  x  ft  with  the  property  that 
G(s,u)  e  L2(ft)  for  every  fixed  s,  and  suppose  further  that 
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0  =  /  G (s ,u) p (u) du,  peL»(ft),  seS 
ft 


(3.1) 


implies  that  p  =  0.  Then  the  range  of  the  operator  G 
defined  by 


(Gp)  (s)  =  /  G(s,u)o(u)du 
ft 


(3.2) 


is  a  reproducing  kernel  Hilbert  space  of  functions  defined 
on  S.  If  we  call  this  space  d  ,  then  it  has  reproducing 
kernel  given  by 


R1(s,s')  = 


f  G (s ,u)G (s Ju) du 

ft 


(3.3) 


with  inner  product 


<f.,f_>  *=  /  t>.  (u)p,  (u)du 
*  ft  * 


(3.4) 


where 


f .  (s)  =  (Gp. )  (s)  =  f  G  (s  , u)  p .  (u) du,  i  =  1,2  (3.5) 

ft 


Suppose  4^  is  contained  in  some  larger  Hilbert  space 
of  functions  defined  on  S,  and  { are  m  orthonormal 
functions  on  S  all  in  .  Then  the  m  dimensional 

space  spanned  by  is  a  reproducing  kernel  space  with 


reproducing  kernel  Rq  given  by 

m 

R  ( s ,  s ' )  =  l  <i  (s)<f>  (s')  (3.6) 

v=l 

and  M  =  Ji  tb  it,  a  -eproducing  kernel  space  with  re¬ 
producing  kernel  R  given  by 

R(s,s')  =  R0(s,s')  +  R^s.s')  (3.7) 

3. 2  Reproducing  Kernel  Spaces  Associated  with  Green’s 
Functions 

Let  L  be  an  mth  order  linear  differential  operator 
in 

on  [0,1]  with  m-dimensional  null  space.  Let  the  linear 
functionals  be  defined  by 

Mf  -  l  v  f (V_1)  (0)  ,  u  =  1,2, ...in  (3.8) 

p  v-1 

where  the  matrix  {y  }  is  n-singular.  Let  G(s,u)  be  the 

Green's  function  for  the  equation  L  f  =  g,  M,  £  «=  0,  u  =  l,2,...m, 

m  y 

and  let  span  the  null  space  of  Lm  with 

M  <£  ~  6  (3.9) 

M  v  MV 

where  <5^v  is  the  Kronecker  6.  We  may  take  as  the 
collection  {f :  Lmf eL^  [0,1]  ,  M^f  =  0,  y  =  l,2...m,}  and  Wg 
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the  space  spanned  by  {<frv>™sl-  #  a  ^  then  has  inner 

product 


<£  .£  >  »  l  (Mvf1)(Mv£2)  +  /  <3'10»> 

v»l  o 


and 


I pl£ I  1 2  -  '  «V»sds 


(3.10b) 


Two  simple  examples  are: 


a) 


_d* 
Jm  "  dsm 


GjS  fU) 


Vs) 


/  .m-1 

(s-uU, 

(m-1)  l 


sv-1 
<v-l)  ! 


i  v  c  1,2... m 


(3.11) 


y  =  1 ,  u  =  v ,  =  0  otherwise 


m 


b)  L, 


m 


G  (s,u) 


n  (D+f*v)  f  {a  }  x  distinct  positive  real 
Vs  1 

numbers 


m  -o  (s-u)  , 

>  3  1  u'  cv  =  " 


l  c  « 
v=l 


otherwise 


(3.12) 


Vs) 


cve 


■°v8 


M  f  =  H  (D+a.) 
v  £/v  * 
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where  (D+ov)f  -  f'  +  avf.  This  type  of  reproducing  kernel 
space  may  be  used  to  construct  spline  functions.  (See  (3]) 
3. 3  Spaces  of  Functions  with  Rapidly  Decreasing  Fourier 
Transform 

Let  W(X)  be  a  complex-valued  Hermitian  function  of 
X  on  (-«,“)  with  W(X)eL2 (-«,«)  and  | W (X ) |  >  0,  all  real  X  . 
Let  G(t)  be  the  inverse  Fourier-Plancherel  Transform  of 
W(X)  which  we  write  as 

op 

G  ( t )  »  /  e“iTX  W(X)dX  .  (3.13) 

CO 

Let 

oo  oo 

R(s,s' )  «  I  G(s-u)G{s'-u)du  -  /  e”i(s"t)XW(X)W*(X)dX 

—  00  —CO 

(3.14) 

R(s,s')  is  positive  definite,  and,  since  |W(X) j  >  0, 

OO 

0  »  /  G  (s-u)p (u)du,  -«  <  3  <  a>(  peL-  (3.15) 

—  OO 

implies  p(u)  ■  0.  W  ,  the  reproducing  kernel  space  for 
R(s ,8 ' )  is  the  collection  of  all  functions  f  of  the  form 

CO 

f(s)  **  /  G(s-u)p(u)du  peLjf-8*,®)  (3.16) 

—00 

with  inner  product  (by  Parseval's  theorem),  given  by 
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-  F.  <X)F,(X) 

<f .  » f ~>  *  /— - j r -  dX  (3.17) 

X  1  -®  W(X)W  (X) 


where  F^  is  the  Fourier-Plancherel  Transform  of  f^.  W 
contains  only  functions  whose  Fourier  transforms  F(X)  de¬ 
crease  sufficiently  rapidly  that  F (X)/W(X) el^  (-w,®) . 

For  a  concrete  example,  let  q ,  p  be  non-negative  in¬ 
tegers  with  q  <  p,  ^et  {ev)^_i  be  p  +  q  distinct 

positive  real  numbers ,  let 


P 


p  (x )  =  n 

(iX-o  ) 

y-l 

M 

q 

q (x)  =  n 

V=1 

(iX-Bv) 

(3.18) 

W(X)  =  Q(X)/P(X). 


We  have 

G(t)  =  \  c  e  0ijT,  t  >  0 

u— 1  v 

(3.19) 

*  0 


and 


R(s,s' ) 


6 


v 


P-1 


c  c 
y  v 

fa  +a  T 

p  v' 


(3,20) 


(for  q  -  0,  p  ■  1  we  have  R(s,s')  «  e”al8”8  I). 

Letting  m  »  p-q ,  H  is  here  the  collection  of  all 
functions  with  absolutely  continuous  m-lst  derivative  and 
square  integrable  mth  derivative.  Let 


Gq(t) 


?  t  >  o 

V«1 


0 


where 


n 

j^v 


(Pj-V 


Let 


(GQf)t  -  /  Gq (t-s) f (s)ds 

*  »CO  " 


V1 

(3.21) 


(3.22) 
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and  let 


P 

L_f  «  n  (D+O  ) f  .  (3-23) 

P  M-l  V 

For  fe  w  t  (G^f )  has  p-1  absolutely  continuous  and  p 
square  integrable  derivatives,  and  the  inner  product  in  W 
is  given  by 

<f1'f2>  *  "  <LPGQfl)t(LPGQf2,tdt-  {3‘24) 

—  00 

3. 4  Spaces  of  Band  Limited  Functions 

Let  3  be  the  subset  of  L2(-w,®)  of  functions  whose 
Fourier  transforms  vanish  outside  [ — A  , A ] . 

Let 

oo 

G  (r )  =  /  s (r-u) w (u) du  (3.25) 

—  00 


where 


s  (r) 


sin  At 
ttt 


w(u) 


A 

! 

-A 


iuX 


W  (X )  dX 


(3.26) 


and  W (X )  is  assumed  to  be  a  Hermitian  function  of  X  with  no 
real  zeroes  in  [-A,AJ.  The  equation 


OO 

0  =  /  G(t-u)p(u)du  -»  <  t  <  ®  (3.27) 

—00 
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has  solutions  in  L2 (-»,»)  but  not  in  $A,  since  (3. 25) -  (3. 27) 
imply  w(A)P(X)  -  0,  -A  <_  X  £  A,  where  P(X)  is  the  Fourier- 
Plancherel  transform  of  p(u).  (Note  that  S(X),  the  Fourier- 
Plancherel  transform  of  sin  At/itt  is  S(X)  ■  1,  | X |  <  A,  »  0 
otherwise. ) 

3  is  a  reproducing  kernel  Hilbert  space  with  inner 
A 

product 


A  F.  (X)F,(X) 

<f1(f0>  -  /  — - 5f - -  dX  (3.28) 

1  z  -A  W(X)W  (X) 


where  F^  is  the  Fourier-Plancherel  transform  of  f^,  i  «  1,2. 
The  reproducing  kernel  is 

R(s,s')  -  /  e_iMs"8,)W(X)W*(X)dX  .  (3.29) 

-A 


If  we  set 

W(X)2  -  1,  |  X |  <  A 


«  0  otherwise 


(3.30) 


then 


R(SfS' ) 


sin  A(s-s') 
ir  (s-s' } 


(3.31) 


and  3^  is  a  Hilbert  subspace  of  L2 (-«,»).  Slepian  and 
Pollack  110)  consider  the  space  of  bandlimited  functions  with 
reproducing  kernel  (3.31).  A  restatement  of  the  sampling 
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theorem  tells  us  that  the  functions  rjj 

=  -i-  /  ei<3-2’^A> 

3  /TK  -A 


defined  by 

W(A)dA,  j 


•  •  »""1|0^1|  I  .  . 

(3.32) 


are  a  complete  orthonormal  basis  for 
4 .  Typical  Theorems 

A  variety  of  theorems  now  fall  out  by  applying  Theorem 
1.1  and  the  lemmas  of  Section  2  to  the  reproducing  kernel 
Hilbert  spaces  of  Section  3.  For  concreteness  only  we 
state  them  with  reference  to  the  example  of  Section  3.2, 
since  this  example  provides  a  direct  comparison  with  the 
method  of  regularization  as  discussed  by  Tihonov  and  Glasko. 

Thus,  let  Lm  be  an  mth  order  linear  differential  operator 
on  [0,1]  with  m-dimensional  null  space  spanned  by 
and  «  =  {ziL^zeL^ [0,1] ,  z'  ^absolutely  continuous}.  Let 
G(s,u)  be  the  Greens  function  for  the  equation 

Lmf  =  g'  f(V)<°)  *  °»  v  "  0,1,2,. ..m-1 


ij  The  reader  may  verify  that  the  choice  of  the  matrix 
{Ayv>  in  (3.8)  is  irrelevant  to  the  solution  of  our  problems, 
we  take  it  as  I. 
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and 


(4.1) 


1 

R.(s,s')  »  /  G(s,u)G(a' ,u)du 

A 


m  1 

R(s,s')  »  l  4>  (a)4i  (s')  +  /  G  (B  ,u)G  (s' ,u)du 
v-1  0 

(4.2) 


We  suppose  that  K(t,s)  satisfies  that 


1  1 

/  /  K(t, ,s)R(s,s')K(t. ,  s 1 ) dsds 1  i  ■  1,2, ...n 
0  0  1  1 


is  well  defined  and  finite  as  a  Riemann  integral. 
Let  M  )  be  defined  by 


1 

t\.  (s)  «  /  K(t,  ,u)R(s,u)du  (4.3) 

1  0  1 


Define  the  elements  ^  e  W  ,  and  the  matrices  X  and  £  by 


1 

Pini  *  ^i ‘  ^i(a)  “  £  KU^uJRjfs.uJdu,  i  ■  1,2, ...n, 

(4.4) 
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X 


=  (xpi>/  VI  “  1,2,.  „  -R,  i  -  1,2, ...n 


I  “  3  =  1,2,.  ..n 


1 

Xpi  =  K{ti,u}4)^  (u)du  (4.S) 

11 

0..  =*  <£.,£.>  •  //  K  (t,  ,s)R.  (3, s'  )K(t .  ,s'  )dsds' 

ID  1  J  00  1  3 


(4.6) 


Theorem  4 . 1  Let  u  =  (u-^  ,u2  , .  . .  un)  be  a  vector  of  n 
real  numbers,  let  V  =  (v^}  be  an  n  x  n  symmetric  positive 
definite  matrix  with  V  1  ■  {vi3},  and  suppose  that  matrices 
X  and  l  defined  by  (4.5)  and  (4.6)  he  of  full  rank.  For 

A 

ze  )i  ,  define  the  numbers  by 

1 

u.  =  u,  (z)  *  <n- ,z>  =  /  K(t. ,s) z (s)ds  (4.7) 

ii  l  o  1 

i  —  1,2, . . ,n 

Then 

A 

(i)  There  is  a  unique  solution  zq  to  the  problem: 
Find  ze  it  to  minimize 


M(z) 


n  .  •  1 

y  (u. -u. )v  3 (u .~u , )  +  a  / 

i ,3=1  11  33  0 


a-  z)  2ds 
m  s 


(4.8) 
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given  by 
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(4.12) 


subject  to  (z)  *  u^,  i  «=  1,2,  ...n. 

(iii)  For  any  zc  J*  satisfying 

1 

u.  -  /  K  (t .  ,s)  z  (s) ds  i  »  1,2,  ...n 
0  1 

A 

and  zQ  given  by  (4.11)  we  have  the  pointwise  error  bounds 
|zo(s)-z(s)  |2  <  ^2<s)  (/(Lmz)2  ds}  (4.13) 

where  a- 

o2(s)  *  R^s.s)  -  ?(s)  I'1  €(s)’  (4.14a) 

+  ($<s)-£(s  )l~1X')  (xf^M'^Hsl-KsjfV)' 

<4. 14b) 

(iv)  If  ze  W  has  the  form 


<)>(s)  =  (<ji1(s)  ,<|»2(s)  ,•..♦„(»), 
€(s)  =  (^(s)  ,C2(s)  ,...5n(s)) 


z(s)  =  l  C  ♦  (8)  +  l  djL?i(s) 

\>=1  i=l 

where  d  =  (d^,d2 , . . .dn)  is  any  vector  satisfying  Xd' 
and  we  set  in  (ii)  as 


(4.15) 

“  0, 
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(4.16) 


1 

u.  =  /  K(t. ,s)z(s)ds 
1  Q  1 

Aw 

then  the  solution  zQ  in  (ii)  satisfies 

■  z  (4.17) 

o 

Proof :  Assertion  (i)  is  Lemma  2.7  and  (ii)  is  Lemma 

2.1.  The  remarkable  error  bound  of  (iii)  is  obtained  from 

Lemmas  2.5  and  2.6  upon  setting  6q  as  the  representer  of 

the  continuous  linear  functional  N  defined  by  N_z  *  z(s), 

s  s 

ze  *  ,  that  is,  6  -  R  .  In  this  case  we  have 

o  s 

A  A 

|zo(3)-z(s)|  -  l<20-z,Rs>| 

<R„,4>  >  ■  ♦„<»)  (4.18) 

3  M  M 

<Rs.Ci>  »  ^(s) 

^iv)  is  mother  way  of  writing  Lemma  2.3. 

Other  minimization  problems  may  be  handled  within  the 
context  of  the  geometry  of  reproducing  kernel  Hilbert  space. 
For  example,  the  solution  to  the  problem:  Find  zs  M  to 
minimize  M(z)  of  (4.8)  subject  to  the  linear  inequalities 

a^  <  z(sA)  _<  b^  ,  l  -  1,2, ...k  ,  s^eS  (4.19) 
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may  be  reduced  to  a  standard  problem  of  minimizing  a  (finite) 
quadratic  form  subject  to  linear  inequalities,  as  follows: 
Inequalities  (4.19)  may  be  written 

a#  ^  ,2^  ^  b, ,  1  **  1,2, ...k  (4.20) 

x  ”  S1  “  * 

Then,  any  solution  z  to  this  minimization  problem  must 
be  of  the  form 

z  "  1  cv*v  +  X  diCi  +  X  e*Rs  <4*21) 

V«1  v  i»l  i  i  1=1  *  s£, 

for  some  coefficients  {cv>,  {d^>  and  {e^}.  By  substituting 
3<4.2l)  into  (4.20)  and  (4.8),  the  minimization  problem  re¬ 
duces  to  the  standard  problem  of  minimizing  a  quadratic  form 
in  the  coefficients,  subject  to  a  set  of  linear  inequalities. 
This  minimization  problem  may  be  recognized  as  a  problem  in 
control  theory.  See,  for  example  [5]. 

Side  conditions  which  are  just  sufficient  to  specify 
PqZ  give  especially  simple  looking  answers  and  error  bounds. 
For  example: 

Theorem  4.2.  Under  the  assumptions  of  Theorem  4.1: 

(i)  The  solution  to  the  problem:  find  ze  #  to 
minimize  (4.8)  subject  to  the  boundary  conditions  fi 

M  z  =  <<b  ,z>  «  0  ,  v  =  1,2... hr  (4.22) 

u  V  V  V 
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is  unique  and  is  given  by  z a, 


2  -  4>e*  +  Sr”1  (u-X’S' ) 

a  u 


(4.23) 


where  9  f ®2' *  * 

A 

ii)  lim  z  -  zo,  where 
a+O 


►e*  +  5rl<S-x*V> 


(4.24) 


is  the  solution  to  the  problem:  Find  ze  »  to  minimize 


/  (Lm2)«ds 
0  m  s 


(4.25) 


subject  to 


f  K (t .  ,s)z(s)ds, 
0  1 


i  -  1,2, ...n 


(4.26) 


Myz  ■  6V»  v  «  1,2,.. .m. 


(4.22) 


(iii)  If  ze  #  satisfies 


u.  «  /  K(t4 ,s)z(s)ds 
1  0 


(4.27a) 


Myz  «  9V,  v  «  l,2,...m 


(4.27b) 
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(4.28a) 


then  zq  of  (4.24)  satisfies 


| 2  (s)-sQ(s) | 2  <  a2 (s) {/ (Lmz)ydu) 

where 

o2(s)  -  Rx  (s , s )  -  l(s)i"15(s)' 


<4. 28b) 


and  £(s)  is  given  by  (4.14b). 

Proof:  ze#,  Mvz  -  6V,  v  -  1,2,... m  imply  that 


V 


m 

l 

v=l 


9  d> 
vrv 


and 


1 

I  K  (t.  ,s)  z  (s)ds  «  <f|,  ,  z> 
0  1  1 


<Ci,P1z>  + 


m 


v»l 


0  Y  • 
V*Vi 


(4.29) 


In  this  case  (4.8)  may  be  written 


+ 


(4.30) 


Since  and  P^  are  in  »  we  may  find  (P^)  to 

minimize  (4.30)  or  (P^z)  to  minimize  (4,25)  subject  to 

#990 


m 

wA  -  I  6VXU<  ■  <C1.P1«>.  i  “  1,2,. ..n 
v™l  Vi  A  x 

via  Lemmas  2.7  and  2.1  respectively,  by  setting  m  ■  0, 
ft  =  W  1  in  these  lemmas.  To  prove  (iii)  let  be  the 
projection  operator  onto  the  subspace  spanned  by 
If  zc  ft  satisfies  (4.27a)  and  (4.27b),  then 


P^z  «  PjZ0  (4.31a) 

V  ■  Vo  ,4-3lb) 


and  hence 


2  -  Z. 


<VV*' 


But 

A  A 

Z(s)-z„(s)  ■  <z-z,R> 

o  O  S 

"  <(P1-Pc)z,Rg> 

*  <(P,-Pr)z,(P.-Pr)R  > 
j.  c,  x  ^  s 


(4.32) 


(4.33) 


Hence 


|z(s)-zo(s)  |< 


||(P1-Pc)z||||p1-Pc)Rs||  <  | |P1z|| || (p1-Pe)Rs|| 


(4.34) 


-38- 


#990 


But 


o2(s)  =»  I  I  (P1-Pc)Rg|  I2  (4.35) 

A  direct  comparison  with  the  solution  of  the  methods 
of  regularization  and  statistical  estimation  obtains,  as 
follows : 

Set  V*1  «  D  in  (4.8)  where  D  is  the  diagonal  matrix  of 
(1.7),  and  set  6  =  0  in  (4.23).  Choose  s^e[0,lj, 

as  in  Section  1  and  let 


u  =  (u^Uj  , . .  .un)  ' 

be  a  given  vector  of  real  numbers  and 

z  *  (z^,^2'  * '  '*)?) ' 


where 


Then 


where 


A  A 

ii  =  Vs!5  '  za  = 

z  =  E([+aD~1)"1u 

is  the  k  x  n  matrix  with  i, 
1 


cr 


entry 


C^(s.)  =  /  K(t.,u)R. (s. ,u)du 
J  i  0  J  ±  1 


(4.36) 


(4.37) 
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and  l  is  the  n  x  n  matrix  with  i,  jth  entry 
1  1 

<£* ,£4>  ■  /  /  K(t.  ,u)R.  (u,u')K(t.,,u')dudu'  (4.38) 

A  J  0  0  3 

If  are  a  set  suitably  chosen  quadrature  co¬ 

efficients  ,  then  we  have,  for  purposes  of  comparison, 
approximately 

k 

^j(sA)  *  l  Mtj  ,u£)R1(s1,uA)ojl  (4.39) 

k  k 

<5i'V  “  rli  K(VVRl<VV,K(tj'V‘YV 

(4.40) 

Let  K  be  the  n  x  k  matrix  (of  (1.7)),  with  i,  jth  entry 
K(tj,s^)o^,  and  R  be  the  k  x  k  matrix  with  i,  entry 
Rl<si ,s j) •  Then  we  may  write  (4.36)  as 

z  *  RK (KRK'+aD"1)"1^  (4.41) 

We  may  identify  R  with  Q  ^  of  section  1  as  follows: 

Suppose  fe  then  if  I^f  ■  g,  we  have 

1 

f(s)  =  I  G(s,u)g(u)du  (4.42) 

0 
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and 


/  (L  f)2ds  =  /  g2 (s) ds  (4.43) 

0  m  0 

Let  f  =  (fts.^,  f  (s2)  ,...f  (sk))  '  ,  g  =  (gts.^,  g (s2) , . . .g  (s^) ) ' , 
G  be  the  k  x  k  matrix  with  ij^  entry  G(s^,Sj)dj  and  Dq  be 
the  k  x  k  diagonal  matrix  with  i,  i1*1  entry  o-^  . 

Since 


1 

R(s,s')  =  /  G (s ,u) G (s ' ,u) du,  (4.44) 

0 


we  have 


R  a  GD_1G ' 
a 


(4.45) 


also 

f  *  Gg  (4.46) 

f'Qf  *  /  (L  f ) 2ds  =  /g2 (s)ds  *  g'D  g  a  f ’G-1 ' D  G_1f  «  f’R*1f 

q  m  s  o  °  ° 

(4.47) 


Some  questions  of  convergence  may  be  answered  as  follows: 
For  simplicity,  we  consider  that  the  boundary  values  of 
the  solution  z  to  (1.1)  are  known.  We  may  then  consider, 

A 

without  loss  of  generality  that  z  and  zq  are  in  tt 


9 
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Let  the  family  of  functions  0  £  t  <  1  be 

defined  by 

1 

£*.(8)  »  ;  K(t,u)R.  (s,u)du  0  <  t  <  1  (4.48) 

c  0  1 

If  the  family  of  functions  {£t  ,  0<t<l}  span  the 

2 

(separable)  space  then  a  (s)  of  (4.28)  ,  will  tend  to 

0  for  each  s  as  the  set  {t^)"ml  becomes  dense  in  [0,1],  by 
(4.35).  A  necessary  and  sufficient  condition  that 
{£t,  0<t<l}  span  is  that 

<£t,z>  ■  0  0  <  t  <  1,  *e  (4.49) 

implies  that  z  -  0.  But  (4.49)  may  be  rewritten 

1 

/  K(t,s)z(s)ds  «  0,  0  <  t  <  1,  ze  tl,  (4.50) 

0  ~ 

imples  z  =*  0.  There  does  not  seem  to  be  a  straightforward 
general  way  of  establishing  a  rate  at  which 

o2(s)  -  |  |  (P^P^Rj  I2  (4 •  35) 

tends  to  zero,  as,  say  sup|t^+^-t^|  -*•  0,  if  indeed  such  a 
rate  exists.  However,  results  have  been  obtained  regarding 
the  convergence  of  ||(P1-P^)6||  when  6e  # 1  is  "very  smooth". 
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Thus  error  rates  for  the  pointwise  approximation  of  very 
smooth  z,  or  for  the  approximation  of  continuous  linear 
functionals  with  very  smooth  representers,  are  availab  e. 

These  results  will  appear  separately. 

5.  The  Introduction  of  Quadrature  Formulae  ar.d  the  Choice  of  a 
If  the  integral 

1 

(s)  =»  /  K  (t,  ,u)R.  (s,u)du  (4.4) 

x  0 

can  be  evaluated  analytically  at  values  of  s  for  which  it 
is  desired  to  estimate  z(s),  and  xy^  and  of  (4.5)  and 
(4.6)  are  known  exactly,  and  computational  and  experimental 
errors  are  negligible,  then  it  is  natural  to  estimate 

A 

35  (s)  by  zQ(s)  of  Theorem  4.1  or  Theorem  4.2.  The  purpose 
of  this  section  is  to  study  the  situation  where  (cr.  .} 

r*  “■  ■**  J 

and  {£.  (s)  ,  must  be  evaluated  by  quadratures,  where 

quadrature  error  is  the  primary  source  of  error.  Let 
{s*}".,,  Sye[0,l],  and  be  suitably  chosen  quadrature  points 

and  quadrature  coefficients,  respectively.  We  show  that  this 
situation  leads  in  a  natural  way  to  estimating  z  by  the  solution  to  the 
problem:  Find  ze  W  to  minimize 

n  N  -  1  , 

M  (z)  =  l  (u.-  I  K(t.,sk)a»kz(sk))4  +  X  /  (^z)  ^ds 
i=l  k=l  0 

(5.1) 
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where  X  is  chosen  to  approximate  the  mean  square  quadrature 

A 

error.  Define  e  #  by 


A 


X  K<ti-sk>“kR,k 


i  =  1 , 2 , . . .  n 


(5.2) 


Lettinq  R,  e  be  that  element  of  W  whose  value  at  s 
1,Sk  1 

given  by  Rj^  g  (s)  =  Ri*s'sk^'  k  =  1,2,...N,  define  ^ 

by 


N 


5i  “  ^  K(ti'sk)wkRl,sk  '  1  =  1 / 2  , . . ,n 


(5.3) 


A  A 


Then  P1ni  «■  i  *  l,2,...n.  xui  given  by 


a  *  W 

*pi  *  ‘W  *  K(ti’Sk,"kVSk> 


(5.4) 


is  a  quadrature  approximation  to  t  j  given  by 


N  N 


’«  ‘  ‘W  °  ll1  K<tt'sk>“kRi<sk'3t>K(tj'sn 


ho. 


(5.5) 


A 

is  a  quadrature  approximation  to  and  £^(s)  given  by 


?i(s) 


N 

J.  K(ti-V“kV8'V 


(5.6) 


is  a  quadrature  approximation  to  £|^  (s)  of  (4.4),  for  each  s. 
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(5.7) 


Define  e .  e  &  by 

1 


e .  = 


=  n±-n± 


The  problem  may  now  be  viewed  as  that  of  approximating 

A 

z(s)  =  <P.  ,z>  from  the  information  u.  =  <n.  +e.,z>,  i  =  1,2,. 

S  XXX 

/V 

or,  alternatively  approximating  Rg  by  ^i«i »  w^ere 

are  unknown. 

Let 


y 


1  di<'VEi) 

i=l  1  1  1 


where  d  =  (d^  ,d£  >  •  •  •  ^n)  are  to  be,  found  so  that  y  is  a  good 
approximation  to  R,,.  If  we  try  to  choose,  d  po  minimize 
||R  — y | |  in  the  error  bound 

S 

t  ■  .  ■  ; .  ' 

|<z,Rg-y>|  <  | ! z J |  | | Rg-y I  I  (5.8) 


it  is  necessary  to  know  <R  ,  e.>  =  e.(s),  i  =  1,2, ...n, 

si  j. 

whioh  is  assumed  unknown. 

We  will  choose  d  subject  to  the  constraint 


1-1 
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Then 


! !  Vy!  I2  ±  2{l  lp1<Vi|1di**i)  1 12  + 


(5.9) 


Let  V  be  the  matrix  with  i,jth  entry  v±.  »  <eiPGj>  given 


<Ei'ej> 


xx  « 

f  K  (t .  ,u)du{  /  K(t.  ,v)R(u,v)dv  -  £  K  (t .  ,sk)R(u,s.)  w.} 

0  1  o  3  k«l  3 

N  1  N 

-  7  K(t.,s.){/K(t,,v)R(s  ,v)dv-  l  K(t,,s  )R(s  ,sk)wk} 

1  *'  o  3  k*=l  J 


(5.10) 


and  let  X  and  £  be  the  n  x  m  and  n  *  n  matrices  with  entries 

A  A 

^Xyi^  and  respectively,  given  by  (5.4)  and  (5.5). 

By  Lemma  2„8  with  $c)  *  Rg ,  the  solution  Rg  to  the 
problem:  Find  y  of  the  form 


y  3  l  <Mni+ej> 


(5.11) 


to  minimize 


nvvjAVi'2  ■ 


(5.12) 


\ 


subject  to 


I’o'V.IVi'll 

1=1 


(5.13) 


is  given  by 


■'  A  «  <1  A  A  •  A 

Rc  »  i  (s)  (xs'xx' )"  xs"A(n+e) 


+  f,  (s)(s  -s  ^(xs  Xx * )  1s  *■)  (n+e) 


(5.14a) 


where 


and 


4>(s)  =  (4>1  (s)  ,<J>2  (s)  .  •4»m(s) ) 

5(8)  =  (^(s)  ,c2(s)  ,...?n(s)) 

A 

S  -  z  +  V 

A  A  A  A 

n+e  =  (n1+e1,n2+E:2, . .  .nn+en) 


(5.14b) 


Any  "optimal"  approximation  to  R  will  clspsnci  on  the  "un~‘ 
known  V.  Thus  it  is  desirable  here  to  approximate  V  using 
whatever  information  is  available.  A  plausible  approximation 
is 


V  *  XI  (5.15) 

where  X  is  a  "guestimate"  of  the  mean  diagonal  element  of  V, 


£• 
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X 


(5.16) 


l  n 

i  l 


<e. ,e,>, 


i“l 


this  "guestimate"  being  based  on  (5.10)  with  i  «  j  and  the 


properties  of  the  quadrature  formula  being  used. 

Let  Rg^e  be  given  by  (5.14)  with  V  replaced  by  XI, 
that  is, 

A  A  »  A  *  A  A  <  A 

Rs,x  ■  ♦(“)  (XS~1X’)'iXS"A(n+e) 


+ 


A  n  A  1  A  A  A  *  A 

5(s)  <S~A-S~AX(XS~AX'  ) 


1^-1 

AsxA)  (n+e) 


where 


l  +  XI. 


(5.17) 


A 

Then  an  estimate  zx  for  z(s)  is  defined  by 


zx(s)  *  <z'RSfx> 


with 


|z(s)-zx(s) |  <  | | z | | | | Rs”Rg f\ 
The  function  z^  defined  by  (5.18)  is  in  J* 
zA  »  ♦(XS"AX')XS"Au 

A  A  «  A  »  A  A  A  <  1  A  « 

+  5(SX-S"AX(XS"AX')''  S“A)U 


(5.18) 


and  may  be  written 

(5.19) 
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where  <p  and  £  are  vectors  of  elements  of  M  with  <f>  as  in 
( 4  • !))  and 


€  “  ' ^2 "  *  * ^n* 


with  £.  given  by  (5.3).  By  Lemma  2.7,  z,  is  the  solution 

1  A 

to  the  problem:  Find  ze  *  to  minimize 


l  (u.-<n • ,z>) 2  +  X  /  (L  z ) ^ds  (5.20) 

i=l  1  1  0  m  s 


where 


<ni,z> 


N 

K(ti'sk)<Jkz{sk>' 


i  ■■  1,2, ..  .n 


and  X  is  of  the  order  of  magnitude  of  the  mean  square  quadrature  error. 
Thus,  if  the  primary  source  of  error  in  forming  a  com¬ 
putational  estimate  of  z(s)  is  quadrature  error,  then  this 
shows  that  an  appropriate  choice  for  the  regularizing  para¬ 
meter  a  of  (1.6)  is  as  X,  an  estimate  of  the  mean  square 
quadrature  error,  as  defined  by  (5.10)  and  (5.16). 

We  have  not  mentioned  the  choice  of  quadrature  formula. 

M 

Once  the  quadrature  points  (sk)k_j  are  chosen,  the  choice  of 
best  formula  in  the  sense  of  Sard  is  equivalent  to  approxi¬ 
mating  the  element  W  by  a  linear  estimation  of  the 
elements  (R  .  The  optimum  coefficients  in  this  case 
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are  readily  seen  to  depend  on  the  unknown  integrals.  Hence 
a  convenient  quadrature  formula  which  allows  a  "guestimate" 
of  X  should  be  used. 

As  is  widely  known,  as  soon  as  there  are  experimental 
or  computational  errors,  there  is  a  point  of  diminishing  re¬ 
turns  in  choosing  n  too  large.  If,  e.g.  K(t,s)  is  continuous 
then  |  IF"!!  and  III”1!  I  +  “  as  n  becomes  large,  where  IMI 
is  the  spectral  norm.  We  will  quantify  this  statement  and 
indicate  a  mitigating  technique.  To  simplify  the  equations, 
we  let  M  a  «  0,  v  -  l,2,...n».  Then  we  may  let 


U.  **  <£ . +E . 

ui  si  1 


.  ,2> 


1 

f  K (t  • ,s)z(s)ds,  i 
0  1 


1,2 


.n 


(5.21) 


where  are  9iVen  <5*3)  and  ei  *  5i’5i*  Then'  by 

Theorem  4.2  the  solution  to  the  problems  Find  ze  W  toT 

minimize 


<5.22) 


A 

is  given  by  z^. 


/v  A  ^  ^ 


(5.23, 


and 


2 

I  z  (s)-zA  (s)  |  ®  ^Z-Z^.Rj  g>| 


<B'Rlis"Rl,sfX>l  i  N*NMri,b-ri,s,x 


(5.24) 


where 


Vs.A  ’  J,  W‘i> 


with  d  =  (d^ ,dj , . . - dn )  given  by 


d  =  £(s)  (J+XI)-1 

A 

with  £(s)  given  by  (5.14b). 

Now 


R.  -rt  , ! 
l,s  1 ,  s  ,  A 


-  2(!lRi,s-,I,di«ill2+llaieiM2» 


(5.25) 


If  V  is  the  matrix  with  i,  entry  v^,=  <e^,ej>,  then  the 

term  in  brackets  in  (5.25)  may  be  expanded  as 
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R^s.s)  -  2^(s)  ([+XI)“1C(«)‘  +  Us)  ([+XI)'1f  (f+Xl)_:LUs)' 


+  c(s)  (X+xiTMe+x I)  1C (s)  ' 


(5.26) 


and  some  algebraic  manipulation  gives  that  (5.26)  is  equal  to 


A  a  1  A  A  A  "l  / n  A  •  ia  A 

Rjts^)  -  5  (s)  £  i^{s)'  +  5(s)J  ^^(s)1  (5.27) 


1/2  *^*“1 
where  £  '  is  the  symmetric  square  root  of  \  and 


A  =  ([+XI)~1(J1/2vf1/2+X2I) (f+XI) 


(5.28) 


Letting  P£  denote  the  projection  operator  onto  the  subspace 

A 

spanned  by  >  we  have 


I  lp^R1(Sl  I2  =  ?(s)r1€'(s) 


(5.29) 


Lines  (5.25),  (5.27),  (5.28)  and  (5.29)  yield  the  bound 


HRl,s-Rl,s,x1l2  1  2[liRl,s-piRl,s*i2  + 

+  llp|Ri>sil2  •  II (E  +  XI)"1  II  •  llv  +  \l|| 


(5.30) 


If  V  =  XI,  the  term  in  curly  brackets  in  (5.30)  can  be  replaced 


by  X| | (I+XI)"1! 


In  practice  as  N,  the  number  of  quadrature 


points  becomes  large,  one  expects  | |V| |  to  decrease  to  a 
finite  limit  imposed  by  round  off  errors.  As  n(n<<N),  the 
number  of  data  points,  becomes  large,  | ! g“p£Ri t g I  I  <*e“ 
creases,  however,  \ | | (£+AI)-1| | 2  will  increase  for  \  bounded 

A  _  *1 

below  since  |  |  (£)  |  |  •+■  ™  as  n  -*■  ®. 

Let  H  »  {h. . }  be  a  p  x  n  matrix  of  real  numbers  of 

Xt  1 

rank  p.  Let 


a  n  a 

h  “  Jl  hnh 


e 


n 


i=l 


h£iei 


(5.31) 


n  n  1 

u„  =  y  h. . u.  =  y  h..  /  K (t . ,s) z (s) ds ,  H  —  1,2, ...p. 
i=l  511  1  i=l  0 


If  we  estimate  z(s)  assuming  the  information  zs M 

A 

<z,5^+efc>  =  u^,  l  ~  1,2,  ..p 

A 

A  A 

as  before,  only  replacing  the  set  bY  ^£,^£=1  an<^ 

the  numbers  {u± bY  an  error  bound  analogous  to 

the  right  hand  side  of  (5.30)  is  obtained  of  the  form 


2  ( 


!Ri,s-p 


'Ri  1 1 2  + 

:  1,S' 1 


|  PaR.  |  |2{|  (  (H^H'+XI)-1!  |-|  |HVH'+XI|  | >) 

£  **■  9  S 


(5.32) 
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where  P*  is  the  projection  operator  onto  the  subspace 

K  -  D 

spanned  by  If  the  rows  of  H  are  orthonormal  vectors 

in  Euclidean  n-space,  then  the  term  in  curly  brackets  in  (5.32) 
can  only  be  decreased,  as  compared  to  the  term  in  brackets  in 
(5.20),  while  the  first  term  in  (5.32)  may,  for  large  n, 
and  n-P,  may  not  increase  much,  as  compared  to  the  first 
term  in  (5.30).  If  V  -  XI,  then  the  “optimum''  choice  of 
H  to  minimize  the  term  in  brackets  in  (5.32)  is  to  choose 
the  rows  of  H  as  the  eigenvectors  corresponding  to  the  p 

A 

largest  eigenvalues  of  £. 

6 .  ' Statistical  Estimation'  of  Solutions  to  Integral  Equations 

It  is  far  from  coincidental  that  the  method  of  re¬ 
gularization  and  the  method  of  statistical  estimation  lead 
to  the  same  numerical  solution.  Let  R(s,s')  be  a  continuous 
positive  definite  kernel  on  S  *  S,  and  «  R  the  reproducing 
Kernel  Hilbert  space  associated  with  R.  Let  Z  (s) ,  seS  be  a 
stochastic  process  (i.e.  a  family  of  random  variables  in¬ 
dexed  by  s) ,  with  EZ(s)  =  0  ^  and 

EZ  (s) Z  (s' )  =  R(s,s')  (6.1) 


2j  without  real  loss  of  generality 
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Let  K(t,s)  satisfy  the  assumptions  of  the  corollary  to 
Theorem  1.1  and  consider  the  stochastic  model 

U ( t )  =  /  K(t,s)Z(s)ds  (6.2) 

S 

where  obser\ ations  will  be  taken  on  the  random  variables 
U(t^) ,  i  =  1,2,... n.  U (t)  is  a  well  defined  random  variable 
for  every  teT.  We  have  been  studying  the  (deterministic) 
model 


u(t)  =  /  K(t,s)z(s)ds  (6.3) 

S 

where  zc  WD,  and  the  numbers  u(t.),  i  =  l,2,...n  are  available. 

K  1 

The  puroose  of  this  section  is  to  demonstrate  rigorously  that 
the  same  numerical  solution  to  the  integral  equation  is  ob¬ 
tained  wether  the  true  'solution'  is  considered  to  be  an 
element  ze  or  a  realization  of  a  stochastic  process  Z(s) 
with  covariance  R(s,s'). 

Let  &  be  the  Hilbert  space  spanned  by  the  stochastic 

*. j 

process  (Z(s),seS).  (See  (6J).  is  defined  as  follows: 

All  random  variables  Y  which  are  finite  linear  combinations 
of  the  form 


v  -  I  ajK.,) 


sscs 


(6.4) 


are  in  .  An  inner  product  on  the  linear  manifold  of  all 

o 
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such  finite  linear 


combinations  is 


<Y1,Y2>  “  EY1Y2' 


(6.5) 


and  *  is  the  closure  of  this  linear  manifold  with  the 

z 

given  inner  product.  The  precise  source  of  the  duality  be¬ 
tween  • deterministic*  and  statistical  models  is  the  following 

(well  known)  fact: 

i _ 11..  • _ _ _ 1  M  h 

R 

isomorphism  induced  by  the  correspondence 


U  ig  isometrically  isomorphic  to  *iR  under  the 

21 


(s)  -  R, 


s  ' 


Vsf.S 


(6.6) 


Furthermore,  the  random  variable  Ye  *z  corresponds  to  the 


element  ne  if  anc*  only  if 


eyz(s)  *  <h(rRc>  *  n(s),  seS 


(6 . 7) 


The  family  of  random  variables  U(t),  teT  are  all  in  a z  by 
our  assumptions  on  K(t,s)  and 


U(t)  -  h4 


(6.8) 


where  nt  is  defined,  for  each  t,  by  (1.25).  Let 

be  a  specified  set  of  m  orthonormal  elements  in  * R,  and  let 
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{p  }n  .  be  the  m  random  vaii.ab.les  in  which  correspond  to 

V  V"1  ct 

{<t>  ) i  under  the  isomorphism  induced  by  (6.6).  It  can  then 
be  shown  that  Z(s),  seS  has  a  representation  of  the  form 


Us) 


(6.9) 


where  Z^(s)  is  a  zero  mean  stochastic  process  with 


EZ-^sjZ^s')  =  R^s, s')  =  R(s,s')  -  l  <t>v(s)<t>  (s')  (6.10) 

v=l 

and  Ep^p^  =  6^  u,v  =  1,2. ..m.  Let  Z(s)  be,  for  each  s, 
that  random  variable  in  the  subspace  of  h  z  spanned  by 
{Uft^  >2_2  which  minimizes 


E(2  (s) 


Z(s))2 


(6.11) 


subject  to 


E(Z(s)  -  Z(s)|pv,v  *  1,2,.  .n)  *  0.  (6.12) 

It  follows  from  Lemma  2.2  and  (6.6)  with  the  identifica¬ 
tions  Z(s)  -  6Q,u(ti)  -  r^,  Pv  -  <t»v,  and  /  K(t^,s)  Z^  (s)ds  -  ^ , 
that 
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Us)  -  (f  (s)  (X[_1X' )_1Xj;_1U' 

+  Us)  (2"1  -  J^XIX^X'  )-1xJ"1)U'  (6.13) 


where 


4»  (s)  =  (<)»1(s),  4>2  (s)  » . .  .$m(s)) 

Us)  -  (^1(s),  52(s),...5n<s)) 

X  and  £  are  as  in  (2.1)  ,  and  U  is  the  vector  of  random 
variables  given  by 


U  *  (Utt^  ,  U(t2)  ,...U(tn)) 


Thus,  the  numerical  value  Z(s)  based  on  the  model  (6.2) 
and  a  "realization"  U(t^)  **  u^,  i  *  l,2,...n,  is  exactly  the 

A 

same  as  the  numerical  value  of  zQ(s)  of  (4.11),  based  on 
the  model  (6.3).  An  identical  statement  may  be  made  about 
z^(s)  of  (5.19)  if  we  replace  (6.2)  by  the  stochastic  model 


u(ti) 


N 

l  K  ^i^k^Jc2  +  6  (fci)  *  =  1*2 
k*=  1 


n (6 . 14) 
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where 


E  i  (ti)  =  0 

E  f  (t^)  <■  (t . )  =  0 ,  i  +  j 

=  \  ,  i  =  j 


E  i  (t^Z  (sR)  =  0, 


i  =  1 , 2  , .  .  .  n 
k  =  1,2, ..  .N 


* 

If  we  let  U  be  the  Hilbert  space  of  all  continuous 

•k 

linear  functionals  on  it  R,  then  it  is  consequently 
isometrically  isomorphic  to  Ji  z  under  the  correspondance 

Z(s)  -  Ng 

where  N  is  the  continuous  linear  functional  defined  by 
s 

Nsz  =  z(s)  ,  z i.  ^ 


Then 


U(t.) 


where  Afc  is  the  continuous  linear  functional  defined  by 
i 
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ze  M 


A  z  -  /  K (t. ,s) z (s)ds , 
Ci  S 


R 


It  is  seen  that  the  geometry  for  approximating  Z(s)  by 

{U  (t.. )  is  exactly  the  same  as  the  geometry  for 

approximating  by  {Afc  . 

i 

An  experimenter  approaching  the  problem  with  the  model 
(6.2)  chooses  the  prior  covariance  R(s,s')  of  (6.1)  according 
to  his  belief  or  past  experience  concerning  Z(s).  The 
numerical  analyst,  beginning  with  (6.3)  should  choose  an  R 

such  that  the  norm  of  the  solution  z  in  W  is  known  or 

R 

believed  to  be  small. 

It  is  clear  that  algorithms  for  the  numerical  solution 
of  a  broad  variety  of  (linear)  equations  can,  in  fact,  be 
identified  with  prediction  problems  on  stochastic  processes 
in  this  manner. 
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We  tic.U  the  problem  o'  obtaining  good  approximate  solutions  to 

u(t)  =  |  K(l,s)  z(s)ds,  t  =  t^,  t7.  . .  . ,  t  ,  as  a  problem  in  the  approximation 
S 

ol  one  continuous  linear  I  unction.,  1  in  a  luprodncing  kernel  Hilbert  space,  by 
several  otiios.  the  methods  ot  regular liiation  and  statistical  estimation 
me  shown  to  be  special  oases-,  lointwise  error  bounds  and  a  criteiia 
lor  cnoosing  the  royuluiination  pniameter  are  given.  . 


